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(54) Abstract Title 

Personal object recognition system for visually impaired persons 

(57) A system for identifying living entities and/or inanimate objects , comprising a digital video camera 10 
mounted in a pair of dark glasses 1 2 worn by the user. The digital video camera 10 transmits a digital video 
signal to a portable computer 16 running an image analysis program in which are stored details of the objects 
to be recognised, The system also stores a uniqueaudtble or tactile signal associated with a particular object. 
Upon identifying an object or entity with the field of view the corresponding audible signal is played in stereo 
through ear pieces 18 worn by the user. The present invention provides an object recognition system for use 
by blind and partially sighted people. 
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PT7.f ("MUNITION AND IDENTIFICATION APPARATUS 



Field of the Invention 

This invention relates to apparatus for recognition and identification of living entities and 
inanimate obj ects, and in particular, to apparatus for aiding blind and partially blind people in 
the recognition and identification of such entities and objects. 

Background to the Invention 

It is well known that blind and partially blind people often compensate for their lack of sight, 
at least to some degree, by using their non-visual senses, in particular their senses of touch and 
hearing, to identify living entities and inanimate objects in their surroundings. In addition, 
they often memorise the layout of a room or other environment so that they can move around 
that environment relatively freely without bumping in to any obstacles such as furniture or the 
like. 

However, the sense of touch is only useful for identifying objects or living entities which are 
within the reach of a blind person. Similarly, their sense of hearing is of little use in 
recognising a person, animal or object which is substantially silent. 

Traditionally, blind people have used white canes to extend their reach so that they can detect 
obstacles in front of them up to a distance equal to the length of the cane and the length of their 
arm. However, such devices are of limited use in actually identifying such obstacles. More 
recently, arrangements have been developed which emit ultrasonic waves and use reflections 
of such waves to detect obstacles. These arrangements are adapted to convert the reflected 
waves into audible signals and/or into movements of an electronic cane guide a blind person 
around an obstacle. As such, this type of arrangement operates to detect single nearby 
obstacles which might otherwise pose a hazard to the user whilst walking. However, no means 
are provided to actually identify the obstacle. 

We have now devised an arrangement which seeks to overcome the problems outlined above. 



Summary of the Invention 
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Thus, in accordance with the present invention, there is provided apparatus for identifying 
living entities and/or inanimate objects, and/or locations the apparatus comprising recognition 
means for recognising or determining the presence of one or more predetermined objects, 
entities or locations, first storage means for storing details of one or more predetermined 
entities objects, or locations, the or each of said predetermined entities, objects or locations 
having associated therewith a unique audible and/or tactile signal, means for matching said 
recognised entity, object or location with the corresponding details stored in said first storage 
means, and means for emitting the unique signal associated with the or each matched feature. 

Also in accordance with the present invention, there is provided a method of identifying living 
entities, and/or inanimate objects, and/or locations, the method comprising the steps of 
recognising or determining the presence of one or more predetermined objects, entities or 
locations, storing details of one or more predetermined entities, objects or locations, assigning 
to the or each of said predetermined entities, objects or locations a unique audible and/or 
tactile signal, and emitting the unique signal associated with the or each matched feature. 

Thus, the present invention provides a system for use in particular (but not necessarily) by 
blind and partially blind people, whereby specific objects, living entities and locations are 
recognised and identified to the user by a unique audible and/or tactile signal. The living 
entities could be specific people known to the user, or types of people, such as police officers 
and the like. The objects could be specific shops, roads, pedestrian crossings, etc. The 
locations could be specific road junctions, for example. Some types of objects, entities and, at 
least types of locations could be pre-programmed for general use, whereas other objects and 
entities could be programmed into or 'learned' by the system for specific users. Such 
"learning" of new objects/entities/locations and assignment of corresponding signals maybe 
achieved by manual selection from a menu of signals when the object/entity/location to be 
"learned" is present by utterance of a spoken signal to be recorded and used as the signal 
(perhaps until such time as an alternative signal is assigned). 

The recognition means may comprise image capturing means (such as a video camera or the 
like), whereby the storage means stores details of one or more predetermined entities, objects 
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and/or locations, and the apparatus further comprises matching means for determining whether . 
any of the features in images captured by one image capturing means match the stored 
predetermined entities, objects or locations. In another embodiment, the recognition means 
may comprise a global positioning system (GPS), and the storage means may store a map (or 
equivalent). In this case, the apparatus preferably comprises a compass or the like to orient 
the user relative to the map. In yet another embodiment, the objects, entities or locations to be 
recognised may be provided with remotely detectable tag or marker and the recognition means 
comprises detection means for detecting the tag, means being provided for determining the 
object, entity or location in or on which a tag has been identified. In this case, means may be 
provided for transmitting an enquiry signal towards an obj ect, entity or location, the tagbeing a 
marker being arranged to transmit a response signal back, possibly indicating the identity of 
the corresponding object, entity or location. 

In one preferred embodiment of the invention, the system could be arranged to 'find' one or 
more specific objects or entities and only emit those signals associated therewith. For 
example, if the user has arranged to meet a specific person, the system could be arranged to 
search the images captured thereby for that person and emit their associated signal only when 
that person is recognised. Other pre-programmed objects and entities would effectively be 
ignored. Alternatively, the system could be arranged to emit the associated signals for all pre- 
programmed objects and entities as and when they are recognised. The system may provide 
means whereby the user can disable, delay or acknowledge the signals emitted thereby. It may 
also provide means whereby the user can select a 'snooze' function, which has the effect of 
stopping the signal being emitted and restarting it after a predetermined period of time if the 
object or entity associated therewith is still within the field of view of the image capturing 
device. 

In yet another embodiment, the apparatus may be used in a vehicle, to signal, for example, the 
presence and position of a bicycle, pedestrian or other hazard near the vehicle. For instance, 
the apparatus may be arranged to emit a signal which sounds by a bicycle bell seeming to come 
from the direction of the bicycle detected in the driver's blind spot or perhaps behind the 
vehicle. In a further embodiment, the apparatus may be used to warn a cyclist of vehicles 
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appmaching him from behind. In this case, the apparatus may comprise a rear facing image _ 
capture device and audio signal generators) incorporated within a cycling helmet or the like. 

The recognition means is beneficially mounted in a user-wearable device. In one preferred 
embodiment, the image capturing device may be head mounted in, for example, a pair of dark 
glasses or the like to be worn by the user. The video sequence captured by the camera is 
beneficially fed to a portable image recognition and tracking system. 

The system preferably further comprises at least one, and beneficiaUy two, earpieces to be 
worn on or in the user' s ears through which the signals are played in response to recognition of 
a particular object or entity. In one preferred embodiment of the invention, means are 
provided for varying the volume and stereo positioning of the emitted signal to convey position 
and movement of the respective object or entity. Thus, for example, in the case where the 
system is arranged to recognise people for which it has been 'trained', unique signature tunes 
may be played quietly while they are within the field of view of the image capturing device, 
with the volume increasing as they move closer to the user and fading away as they move out 
of the filed of view. The signal may also be arranged to shift from one earpiece to the other as 
a person moves across the field of view of the image capturing device. 

The system may be further enhanced by being adapted to associate specific signals with 
specific locations on a stored map to aid the user in finding their way around. For example, 
when a specific road junction enters the field of view of the image capturing device, the system 
may be arranged to play a specific theme tune, played in a direction (using the earpieces) and 
at a volume determined by the direction and distance of that junction or the next junction or 
landmark on a route, relative to the user, the latter being particularly useful, for example, 
guiding a blind person to a particular locality in an unfamiliar town or for providing a route 
guidance function in a vehicle without the need for visual displays which could distract the 
driver. This information may be obtained by means of a positioning system such as GPS or the 
like. In one embodiment of the invention, an audible signal could be associated with an 
extended object, such as a selected route through a building or a long distance footpath, the 
system preferably being arranged to vary the strength of the signal so that it becomes stronger, 
say, as the user of the apparatus strays away from the selected route. 



Brief Description of the Drawings 

An embodiment of the invention wiU now be described by way of example only and with 
reference to the accompanying drawing which is a schematic block diagram of an exemplary 
embodiment of the present invention. 

Detailed Description of the Invention 

Referring to Figure 1 , apparatus according to an exemplary embodiment of the present 
invention comprises a digital video camera 1 0 mounted in a pair of dark glasses 12 worn by a 
user. The digital video camera 10 transmits a digital video signal to a portable computer 16 
{by means of a hard-wired connection or a wireless connection such as Bluetooth or the 
like), the portable computer 1 6 running an image analysis program in which is stored details of 
aplurality of different objects and living entities required to be recognised, together with then- 
associated unique audio signals (such as tunes). 

The image analysis program may be chosen from, or may utilise, a number of conventional 
image recognition programs suitable for the purpose. One of the more difficult recognition 
problems is that of face recognition and identification - examples of appropriate face 
identification systems will now be discussed. A leading example is the MIT face recognition 
system developed by the Vision and modeling group of the MIT Media Lab, described at 
http:/Avww-white.media.mit.edu/vismod . 

Examples of existing software which is able to identify a face from an image is as follows: 

Beyond Eigenfaces: Probabilistic Matching for Face Recognition Moghaddam B., Wahid W. 
& Pentland A. International Conference on Automatic Face & Gesture Recognition, Nara, 
Japan, April 1998. 

Probabilistic Visual Learning for Object Representation Moghaddam B. & Pentland A Pattern 
Analysis and Machine Intelligence, PAMI-19 (7), pp. 696-710, July 1997 



A Bayesian Similarity Measure for Direct Image Matching Moghaddam B., Nastar C. 
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Pentland A.International Conference on Pattern Recognition, Vienna, Austria, August 1996. 
Bayesian Face Recognition Using Deformable Intensity Surfaces Moghaddam B., Nastar C. & 
Pentland A.IEEE Conf, on Computer Vision & Pattern Recognition, San Francisco, CA, June 
1996. 

Active Face Tracking and Pose Estimation in an Interactive Room Darrell T., Moghaddam B. 
& Pentland A. IEEE Conf. on Computer Vision & Pattern Recognition, San Francisco, CA, 
June 1996. 

Generalized Image Matching: Statistical Learning of Physically-Based Deformations Nastar 
Q, Moghaddam B. & Pentland A. Fourth European Conference on Computer Vision, 
Cambridge, UK, April 1996. 

Probabilistic Visual Learning for Obj ect Detection Moghaddam B. & Pentland A.International 
Conference on Computer Vision, Cambridge, MA, June 1995. 

A Subspace Method for Maximum Likelihood Target Detection Moghaddam B. & Pentland A. 
International Conference on Image Processing, Washington DC, October 1995. 

An Automatic System for Model-Based Coding of Faces Moghaddam B. & Pentland AiEEE 
Data Compression Conference, Snowbird, Utah, March 1995. 

View-Based and Modular Eigenspaces for Face Recognition Pentland A., Moghaddam B. & 
Starner T. IEEE Conf. on Computer Vision & Pattern Recognition, Seattle, WA, July 1994. 

The MIT system includes a face identification component However a separate system purely 
for face detection (without recognition) is the CMU (Carnegie Mellon University) face 
detector http://www.vasc.rixmu.edu/demos/facedemo.htmL A reference to this system is: 
Human Face Detection in Visual Scenes, Henry A. Rowley, Shumeet Baluja and Takeo 
Kanade, Carnegie Mellon Computer Science Technical Report CMU-CS-95-158R, November 
1995, 



The image analysis program searches the received video images for images of the objects and 
living entities stored therein, and tracks these objects and entities within the field of view of 
the video camera 10. At the same time, the tune or other audio signal associated with each of 
the recognised features is played in stereo through a pair of earpieces 1 8 worn by the user. As 
the user gets closer to the recognised features) or the feature(s) get closer to them, the volume 
of the played signal increases. Similarly, as the distance between the user and the recognised 
featured) increases, so the volume of the emitted signal decreases until a feature moves out of 
the field of view altogether, at which point the signal for that feature ceases to be played. 
The locations of such objects/entities maybe associated with a "map" of the surroundings of 
the user such that their positions can be remembered even when they are out of the field of 
view of the camera 1 0. The "map" might be periodically refreshed as the user moves the 
video camera 1 0 around the area. In this case, the respective signals can be generated such 
that they seem to come from objects/entities all around the user, even if they are only 
recognised and their positions detected or updated when the user turns his head towards them. 

In the foregoing specification, the invention has been described with reference to specific 
exemplary embodiments thereof. It will, however, be apparent to a person skilled in the art 
that various modifications and changes may be made thereto without departing from the 
broader spirit and scope of the invention as set forth in the appended claims. Accordingly, the 
specification and drawings are to be regarded in an illustrative, rather than a restrictive, sense. 
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CLAIMS: 

1 . Apparatus for identifying living entities and/or inanimate obj ects, and/or locations, 
the apparatus comprising recognition means for recognising or determining the 
presence of one or more predetermined objects, entities or locations, first storage 
means for storing details of one or more predetermined entities, objects or locations, 
the or each of said predetermined entities, objects or locations having associated 
therewith a unique audible and/or tactile signal, means for matching said recognised 
entity, object or location with the corresponding details stored in said first storage 
means, and means for emitting the unique signal associated with the or each matched 
feature. 

2. Apparatus according to claim 1, including search means for searching images 
captured by said image capturing means and emitting the unique signal associated 
only with a chosen one or more of said predetermined entities or objects. 

3. Apparatus according to claim 1 or claim 2, arranged to emit the associated unique 
signals for all pre-programmed objects and entities as and when they are recognised 
within the images captured by the image capturing device. 

4. Apparatus according to any one of the preceding claims, comprising second storage 
means for storing a plurality of signals for selection and assignment to a 
predetermined object, entity or location, as required. 

5. Apparatus according to any one of the preceding claims, wherein the recognition 
means comprises an image capturing device and image matching means for 
determining whether any of the features in a captured image match said 
predetermined entities, objects or locations stored in said first storage means. 

6. Apparatus according to claim 5, wherein said image capturing device comprises a 
video camera is mounted in a user-wearable device. 
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7. Apparatus according to claim 6, wherein the image capturing device is head mounted 
in, for example, a pair of dark glasses or the like, to be worn by the user. 

8. Apparatus according to any one of claims 5 to 7, wherein images captured by said 
image capturing device are fed to a portable image recognition and tracking system. 

9. Apparatus according to any one of claims 1 to 3, wherein said recognition means 
comprises a global positioning system, and said first storage means has stored therein 
a map (or equivalent). 

10. Apparatus according to claim 9, wherein the objects, entities or locations to be 
recognised are provided with a remotely detectable tag or marker, the recognition 
means further comprising detection means for detecting a tag or marker and 
determining the identity of the respective object, entity or location. 

11. Apparatus according to claim 10, comprising transmitter means for transmitting an 
enquiry signal towards an object, entity or location, the tag or marker being arranged 
to transmit a response signal back to the apparatus. 

1 2. Apparatus according to claim 1 1 , wherein said response signal includes data relating 
to the identify of the respective object, location or entity, 

1 3. Apparatus according to any one of the preceding claims, comprising at least one, and 
more preferably two, ear pieces to be worn or in the user's ear through which the 
signals are played in response to recognition of a particular object or entity. 

14. Apparatus according to any one of the preceding claims, comprising means for 
varying the volume and/or stereo positioning of an emitted signal to convey position 
and/or movement of a respective object or entity. 
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1 5. Apparatus according to any one of the preceding claims, wherein said unique signals 
comprise musical themes or tunes, a different theme or tune being associated with 
each predetermined object, entity or location. 

1 6. Apparatus for identifying living entities and/or inanimate objects and/or locations, the 
apparatus being substantially as herein described with reference to the accompanying 
drawings. 

17. A method of identifying living entities and/or inanimate objects and/or locations, the 
method comprising the steps of recognising or determining the presence of one or 
more predetermined objects, entities or locations, storing details of one or more 
predetermined entities, objects or locations, assigning to the or each of said 
predetermined entities, objects or locations, a unique audible and/or tactile signal, and 
emitting the unique signal associated with the or each matched feature. 

1 8. A method of identifying living entities and/or inanimate objects, and/or locations the 
method being substantially as herein described with reference to the accompanying 
drawings. 



i 




Application No; 
Claims searched: 



Die 

Patent 
I Office 



4- 



TT 



c/: 



INVESTOR EN PEOPLE 



GB 0118599.0 
1 - 18 



Examiner: 
Date of search: 



Mark Gainey 
13 March 2002 



Patents Act 1977 

Search Report under Section 17 

Databases searc hed: __ — 

UK Patent Office collections, including GB, EP, WO & US patent specifications, i 

UKCl(Ed.T): G4N (NAA, NDAX) 
Int CI (Ed.7): G08B (3/00, 6/00), A61H (3/06) 
Other: Online: EPODOC, JAPIO, WPI 



Documents considered to be relevant: 



Category 


Identity of document and relevant passage 


Relevant 
to claims 


A 


GB 2282908 A 


ASH p3. para 2 




Y 


JP 10069539 A 


NIPPON ELECTRIC (see abstract) 


1,17 at 
least 


Y 


US 6055048 


LANGEVIN & MOYNIHAM col. 1 1.64 - col.2 
1.21 col. 3 11.34-55 


1,17 at 
least 



X Document indicating lack of novelty or inventive step 
Y Document indicating lack of inventive step if combined 
with one or more other documents of same category. 

& Member of the same patent family 



A Document indicating technological background and/or state of the art 
P Documentpublished on or after the declared priority date but before the 

filing date of this invention. 
E Patent document published on or after, but with priority date earlier 
than, the filing date of this application. 



An Executive Agency of the Department of Trade and Industry 



